Content generation apparatus and method

ABSTRACT

A content generation system including an image generation unit operable to generate one or more images of a virtual environment for display at a first display device, and an image transmission unit operable to transmit generated images of the virtual environment to each of the first display device and the second display device, where the or each image displayed at one of the first display device and the second display device is a subset of the or each image displayed at the other of the first display device and the second display device, and where the viewpoint of the display device displaying a subset of the or each image is operable to modify the displayed field of view independently of the field of view displayed at the other display device.

BACKGROUND OF THE INVENTION Field of the invention

This disclosure relates to a content generation apparatus and method.

Description of the Prior Art

The “background” description provided herein is for the purpose ofgenerally presenting the context of the disclosure. Work of thepresently named inventors, to the extent it is described in thisbackground section, as well as aspects of the description which may nototherwise qualify as prior art at the time of filing, are neitherexpressly or impliedly admitted as prior art against the presentinvention.

In recent years, the use of head-mountable display devices (HMDs) hasbecome more common in the home for the provision of immersive videocontent. This content may be three-dimensional video content or gamecontent, for example. Immersive content is often considered to bedesirable for a user as it allows them to feel more invested in thecontent, increasing the enjoyment of the user experience. An example ofthis is a virtual reality (VR) experience, in which the object is forthe user to believe that they are present within the virtual environmentthat is presented to them.

One problem that is associated with the use of HMDs is that theexperience is generally limited to a single user. In order to provide animmersive VR experience, it is desirable to generate very high qualityimages and perform tracking of user motion with a reduced latency. Inview of these desires, it is common to leverage as much processing poweras is available to a device generating images for display; in manycases, the available processing power is not sufficient for generatingcontent to enable two or more users to each enjoy a respective VRexperience of a suitable quality.

A large amount of processing power may be required in order to providean immersive experience, and as a result it is not possible to provide a‘lower quality’ immersive experience to two players using a singledevice to generate two streams. This is because a high frame rate andhigh image quality are both necessary for an immersive experience; if anun-responsive and non-smooth viewing experience (high latency/low framerate), or a low image quality video output, is provided then the senseof immersion may be entirely lost by a user.

As a result, only a single user is often able to participate in a VRexperience at a time, which may be undesirable when playing games in agroup setting. One method that has been used in order to mitigate thisproblem is to show the same content as that provided to the HMD on aseparate screen—for example, the left- or right-eye image generated fordisplay by the HMD may also be output to a television or other displayassociated with the processing device. This may allow other people toalso experience the view provided to the HMD user, although theexperience may not be as enjoyable for the viewers of this content. Onereason for this is that the use of a display other than an HMD may meanthe sense of immersion expected of the content is lost; in addition tothis, the viewpoint experienced by the spectators is determined entirelyby the HMD user which may mean that spectators miss out on elements inthe virtual environment that they wish to view.

It is in the context of these problems that the present inventionarises.

SUMMARY OF THE INVENTION

Various aspects and features of the present disclosure are defined inthe appended claims and within the text of the accompanying descriptionand include at least a content generation system and a method ofoperating a content generation system as well as a computer program.

BRIEF DESCRIPTION OF THE DRAWINGS

A more complete appreciation of the disclosure and many of the attendantadvantages thereof will be readily obtained as the same becomes betterunderstood by reference to the following detailed description whenconsidered in connection with the accompanying drawings, wherein:

FIG. 1 schematically illustrates an HMD worn by a user;

FIG. 2 is a schematic plan view of an HMD;

FIG. 3 schematically illustrates the formation of a virtual image by anHMD;

FIG. 4 schematically illustrates another type of display for use in anHMD;

FIG. 5 schematically illustrates a pair of stereoscopic images;

FIG. 6 schematically illustrates a change of view of user of an HMD;

FIGS. 7a and 7b schematically illustrate HMDs with motion sensing;

FIG. 8 schematically illustrates a position sensor based on optical flowdetection;

FIG. 9 schematically illustrates the generation of images in response toHMD position or motion detection;

FIG. 10 schematically illustrates a virtual camera and viewpoint;

FIG. 11 schematically illustrates a change in desired viewpoint;

FIG. 12 schematically illustrates a content generation and displaysystem;

FIG. 13 schematically illustrates a plan view of a virtual environment;

FIG. 14 schematically illustrates a stereoscopic image pair;

FIG. 15 schematically illustrates a stereoscopic image pair withselected image portions;

FIG. 16 schematically illustrates a plan view of a virtual environmentwith multiple viewpoints;

FIG. 17 schematically illustrates a plan view of a virtual environmentwith a viewpoint having a wide field of view;

FIG. 18 schematically illustrates a content generation and displaysystem;

FIG. 19 schematically illustrates an image generation unit;

FIG. 20 schematically illustrates an image generation unit;

FIG. 21 schematically illustrates a content generation and transmissionmethod;

FIG. 22 schematically illustrates a content generation method;

FIG. 23 schematically illustrates a content generation method; and

FIG. 24 schematically illustrates a content display method.

DESCRIPTION OF THE EMBODIMENTS

While the present disclosure refers generally to the use of an HMD asthe primary display device, this is not essential. For example, theadvantages of the arrangement described in the present disclosure may beappreciated when using a 3D television or the like as the primarydisplay device. While the problems described above may not be quite sosevere in such an embodiment, the provision of second video content asdescribed below may still provide a more immersive experience forsecondary viewers.

In some embodiments, the display is a head-mountable display and theposition and/or orientation of the viewer's head is detected bydetecting a position and/or orientation of the head-mountable display.The head mountable display may have a frame to be mounted onto anviewer's head, the frame defining one or two eye display positionswhich, in use, are positioned in front of a respective eye of the viewerand a respective display element is mounted with respect to each of theeye display positions, the display element providing a virtual image ofa video display of a video signal from a video signal source to that eyeof the viewer. In other examples, the display is not a head-mountabledisplay. In some embodiments, the display (whether head mountable ornot) may be referred to as an immersive display, in that in normal useit fills at least a threshold angular range (for example, at least 40o)of the field of view of the user. Examples include multiple projectordisplays, wrap-around (curved) displays and the like.

Referring now to the drawings, wherein like reference numerals designateidentical or corresponding parts throughout the several views, in FIG. 1a user 10 is wearing an HMD 20 on the user's head 30. The HMD comprisesa frame 40, in this example formed of a rear strap and a top strap, anda display portion 50.

The HMD of FIG. 1 completely obscures the user's view of the surroundingenvironment. All that the user can see is the pair of images displayedwithin the HMD.

The HMD has associated headphone earpieces 60 which fit into the user'sleft and right ears 70. The earpieces 60 replay an audio signal providedfrom an external source, which may be the same as the video signalsource which provides the video signal for display to the user's eyes.

In operation, a video signal is provided for display by the HMD. Thiscould be provided by an external video signal source 80 such as a videogames machine or data processing apparatus (such as a personalcomputer), in which case the signals could be transmitted to the HMD bya wired or a wireless connection. Examples of suitable wirelessconnections include Bluetooth® connections. Audio signals for theearpieces 60 can be carried by the same connection. Similarly, anycontrol signals passed from the HMD to the video (audio) signal sourcemay be carried by the same connection.

Accordingly, the arrangement of FIG. 1 provides an example of ahead-mountable display system comprising a frame to be mounted onto anobserver's head, the frame defining one or two eye display positionswhich, in use, are positioned in front of a respective eye of theobserver and a display element mounted with respect to each of the eyedisplay positions, the display element providing a virtual image of avideo display of a video signal from a video signal source to that eyeof the observer.

FIG. 1 shows just one example of an HMD. Other formats are possible: forexample an HMD could use a frame more similar to that associated withconventional eyeglasses, namely a substantially horizontal leg extendingback from the display portion to the top rear of the user's ear,possibly curling down behind the ear. In other examples, the user's viewof the external environment may not in fact be entirely obscured; thedisplayed images could be arranged so as to be superposed (from theuser's point of view) over the external environment. An example of suchan arrangement will be described below with reference to FIG. 4.

In the example of FIG. 1, a separate respective display is provided foreach of the user's eyes. A schematic plan view of how this is achievedis provided as FIG. 2, which illustrates the positions 100 of the user'seyes and the relative position 110 of the user's nose. The displayportion 50, in schematic form, comprises an exterior shield 120 to maskambient light from the user's eyes and an internal shield 130 whichprevents one eye from seeing the display intended for the other eye. Thecombination of the user's face, the exterior shield 120 and the interiorshield 130 form two compartments 140, one for each eye. In each of thecompartments there is provided a display element 150 and one or moreoptical elements 160. The way in which the display element and theoptical element(s) cooperate to provide a display to the user will bedescribed with reference to FIG. 3.

Referring to FIG. 3, the display element 150 generates a displayed imagewhich is (in this example) refracted by the optical elements 160 (shownschematically as a convex lens but which could include compound lensesor other elements) so as to generate a virtual image 170 which appearsto the user to be larger than and significantly further away than thereal image generated by the display element 150. As an example, thevirtual image may have an apparent image size (image diagonal) of morethan 1 m and may be disposed at a distance of more than 1 m from theuser's eye (or from the frame of the HMD). In general terms, dependingon the purpose of the HMD, it is desirable to have the virtual imagedisposed a significant distance from the user. For example, if the HMDis for viewing movies or the like, it is desirable that the user's eyesare relaxed during such viewing, which requires a distance (to thevirtual image) of at least several metres. In FIG. 3, solid lines (suchas the line 180) are used to denote real optical rays, whereas brokenlines (such as the line 190) are used to denote virtual rays.

An alternative arrangement is shown in FIG. 4. This arrangement may beused where it is desired that the user's view of the externalenvironment is not entirely obscured. However, it is also applicable toHMDs in which the user's external view is wholly obscured. In thearrangement of FIG. 4, the display element 150 and optical elements 200cooperate to provide an image which is projected onto a mirror 210,which deflects the image towards the user's eye position 220. The userperceives a virtual image to be located at a position 230 which is infront of the user and at a suitable distance from the user.

In the case of an HMD in which the user's view of the externalsurroundings is entirely obscured, the mirror 210 can be a substantially100% reflective mirror. The arrangement of FIG. 4 then has the advantagethat the display element and optical elements can be located closer tothe centre of gravity of the user's head and to the side of the user'seyes, which can produce a less bulky HMD for the user to wear.Alternatively, if the HMD is designed not to completely obscure theuser's view of the external environment, the mirror 210 can be madepartially reflective so that the user sees the external environment,through the mirror 210, with the virtual image superposed over the realexternal environment.

In the case where separate respective displays are provided for each ofthe user's eyes, it is possible to display stereoscopic images. Anexample of a pair of stereoscopic images for display to the left andright eyes is shown in FIG. 5. The images exhibit a lateral displacementrelative to one another, with the displacement of image featuresdepending upon the (real or simulated) lateral separation of the camerasby which the images were captured, the angular convergence of thecameras and the (real or simulated) distance of each image feature fromthe camera position.

Note that the lateral displacements in FIG. 5 (and those in FIG. 15 tobe described below) could in fact be the other way round, which is tosay that the left eye image as drawn could in fact be the right eyeimage, and the right eye image as drawn could in fact be the left eyeimage. This is because some stereoscopic displays tend to shift objectsto the right in the right eye image and to the left in the left eyeimage, so as to simulate the idea that the user is looking through astereoscopic window onto the scene beyond. However, some HMDs use thearrangement shown in FIG. 5 because this gives the impression to theuser that the user is viewing the scene through a pair of binoculars.The choice between these two arrangements is at the discretion of thesystem designer.

In some situations, an HMD may be used simply to view movies and thelike. In this case, there is no change required to the apparentviewpoint of the displayed images as the user turns the user's head, forexample from side to side. In other uses, however, such as thoseassociated with virtual reality (VR) or augmented reality (AR) systems,the user's viewpoint need to track movements with respect to a real orvirtual space in which the user is located.

This tracking is carried out by detecting motion of the HMD and varyingthe apparent viewpoint of the displayed images so that the apparentviewpoint tracks the motion.

FIG. 6 schematically illustrates the effect of a user head movement in aVR or AR system.

Referring to FIG. 6, a virtual environment is represented by a (virtual)spherical shell 250 around a user. Because of the need to represent thisarrangement on a two-dimensional paper drawing, the shell is representedby a part of a circle, at a distance from the user equivalent to theseparation of the displayed virtual image from the user. A user isinitially at a first position 260 and is directed towards a portion 270of the virtual environment. It is this portion 270 which is representedin the images displayed on the display elements 150 of the user's HMD.

Consider the situation in which the user then moves his head to a newposition and/or orientation 280. In order to maintain the correct senseof the virtual reality or augmented reality display, the displayedportion of the virtual environment also moves so that, at the end of themovement, a new portion 290 is displayed by the HMD.

So, in this arrangement, the apparent viewpoint within the virtualenvironment moves with the head movement. If the head rotates to theright side, for example, as shown in FIG. 6, the apparent viewpoint alsomoves to the right from the user's point of view. If the situation isconsidered from the aspect of a displayed object, such as a displayedobject 300, this will effectively move in the opposite direction to thehead movement. So, if the head movement is to the right, the apparentviewpoint moves to the right but an object such as the displayed object300 which is stationary in the virtual environment will move towards theleft of the displayed image and eventually will disappear off theleft-hand side of the displayed image, for the simple reason that thedisplayed portion of the virtual environment has moved to the rightwhereas the displayed object 300 has not moved in the virtualenvironment. Similar considerations apply to the up-down component ofany motion.

FIGS. 7a and 7b schematically illustrated HMDs with motion sensing. Thetwo drawings are in a similar format to that shown in FIG. 2. That is tosay, the drawings are schematic plan views of an HMD, in which thedisplay element 150 and optical elements 160 are represented by a simplebox shape. Many features of FIG. 2 are not shown, for clarity of thediagrams. Both drawings show examples of HMDs with a motion detector fordetecting motion of the observer's head.

In FIG. 7a , a forward-facing camera 320 is provided on the front of theHMD. This does not necessarily provide images for display to the user(although it could do so in an augmented reality arrangement). Instead,its primary purpose in the present embodiments is to allow motionsensing. A technique for using images captured by the camera 320 formotion sensing will be described below in connection with FIG. 8. Inthese arrangements, the motion detector comprises a camera mounted so asto move with the frame; and an image comparator operable to comparesuccessive images captured by the camera so as to detect inter-imagemotion.

FIG. 7b makes use of a hardware motion detector 330. This can be mountedanywhere within or on the HMD. Examples of suitable hardware motiondetectors are piezoelectric accelerometers or optical fibre gyroscopes.It will of course be appreciated that both hardware motion detection andcamera-based motion detection can be used in the same device, in whichcase one sensing arrangement could be used as a backup when the otherone is unavailable, or one sensing arrangement (such as the camera)could provide data for changing the apparent viewpoint of the displayedimages, whereas the other (such as an accelerometer) could provide datafor image stabilisation.

FIG. 8 schematically illustrates one example of motion detection usingthe camera 320 of FIG. 7 a.

The camera 320 is a video camera, capturing images at an image capturerate of, for example, 25 images per second. As each image is captured,it is passed to an image store 400 for storage and is also compared, byan image comparator 410, with a preceding image retrieved from the imagestore. The comparison uses known block matching techniques (so-called“optical flow” detection) to establish whether substantially the wholeimage captured by the camera 320 has moved since the time at which thepreceding image was captured. Localised motion might indicate movingobjects within the field of view of the camera 320, but global motion ofsubstantially the whole image would tend to indicate motion of thecamera rather than of individual features in the captured scene, and inthe present case because the camera is mounted on the HMD, motion of thecamera corresponds to motion of the HMD and in turn to motion of theuser's head.

The displacement between one image and the next, as detected by theimage comparator 410, is converted to a signal indicative of motion by amotion detector 420. If required, the motion signal is converted by to aposition signal by an integrator 430.

As mentioned above, as an alternative to, or in addition to, thedetection of motion by detecting inter-image motion between imagescaptured by a video camera associated with the HMD, the HMD can detecthead motion using a mechanical or solid state detector 330 such as anaccelerometer. This can in fact give a faster response in respect of theindication of motion, given that the response time of the video-basedsystem is at best the reciprocal of the image capture rate. In someinstances, therefore, the detector 330 can be better suited for use withhigher frequency motion detection. However, in other instances, forexample if a high image rate camera is used (such as a 200 Hz capturerate camera), a camera-based system may be more appropriate. In terms ofFIG. 8, the detector 330 could take the place of the camera 320, theimage store 400 and the comparator 410, so as to provide an inputdirectly to the motion detector 420. Or the detector 330 could take theplace of the motion detector 420 as well, directly providing an outputsignal indicative of physical motion.

Other position or motion detecting techniques are of course possible.For example, a mechanical arrangement by which the HMD is linked by amoveable pantograph arm to a fixed point (for example, on a dataprocessing device or on a piece of furniture) may be used, with positionand orientation sensors detecting changes in the deflection of thepantograph arm. In other embodiments, a system of one or moretransmitters and receivers, mounted on the HMD and on a fixed point, canbe used to allow detection of the position and orientation of the HMD bytriangulation techniques. For example, the HMD could carry one or moredirectional transmitters, and an array of receivers associated withknown or fixed points could detect the relative signals from the one ormore transmitters. Or the transmitters could be fixed and the receiverscould be on the HMD. Examples of transmitters and receivers includeinfra-red transducers, ultrasonic transducers and radio frequencytransducers. The radio frequency transducers could have a dual purpose,in that they could also form part of a radio frequency data link toand/or from the HMD, such as a Bluetooth® link.

FIG. 9 schematically illustrates image processing carried out inresponse to a detected position or change in position of the HMD.

As mentioned above in connection with FIG. 6, in some applications suchas virtual reality and augmented reality arrangements, the apparentviewpoint of the video being displayed to the user of the HMD is changedin response to a change in actual position or orientation of the user'shead.

With reference to FIG. 9, this is achieved by a motion sensor 450 (suchas the arrangement of FIG. 8 and/or the motion detector 330 of FIG. 7b )supplying data indicative of motion and/or current position to arequired image position detector 460, which translates the actualposition of the HMD into data defining the required image for display.An image generator 480 accesses image data stored in an image store 470if required, and generates the required images from the appropriateviewpoint for display by the HMD. The external video signal source canprovide the functionality of the image generator 480 and act as acontroller to compensate for the lower frequency component of motion ofthe observer's head by changing the viewpoint of the displayed image soas to move the displayed image in the opposite direction to that of thedetected motion so as to change the apparent viewpoint of the observerin the direction of the detected motion.

The image generator 480 may act on the basis of metadata such asso-called view matrix data, in a manner to be described below.

FIG. 10 schematically illustrates a virtual camera 500 that may be usedto define a viewpoint in a virtual environment. The position andorientation of the virtual camera 500 may be defined in dependence upona detected position and orientation of an HMD worn by a viewer, by auser input via a controller, or any other suitable method.

The virtual camera 500 may define a viewing area 520 with a viewingangle 530 and a viewing extent 510, the viewing extent 510 representingthe depth to which the viewpoint extends. While shown in two dimensions,it would be apparent that such a viewing area 520 may be extended to avolume so as to define a three-dimensional view.

Images for display may be generated in dependence upon the position andorientation of the virtual camera 500, such that any elements in thevirtual environment that appear within the defined view area/volume aredisplayed to a viewer.

FIG. 11 schematically illustrates a change in the desired viewpoint of auser.

A difference in location of the virtual camera 500 and a viewer's eye540 may arise when there is a non-negligible latency in the tracking orrendering processes, for example, and the viewer moves their head. Sucha movement may be any rotation or translation.

The user, instead of viewing an area 550 as expected, may still bepresented with a view of the area 520 in FIG. 10. This may cause aviewer to lose a sense of immersion, as their view in the virtualenvironment does not match that which would be expected. As a result, asatisfactory VR experience may not be provided to the viewer.

Problems such as this may be magnified when providing multiple userswith their own VR experience using the same hardware; the sharing of theprocessing power results in an increased rendering and/or trackinglatency. As discussed above, this often results in VR experiences beinglimited to a single user only.

FIG. 12 schematically illustrates a content generation and displaysystem 1200 comprising a video source 1210, an HMD 1220 (which functionsas a first display device), and an additional (second) display device1230.

The video source 1210 may be any device that is operable to output videocontent to one or more display devices. The video source 1210 may alsobe able to generate the content, although in some embodiments thecontent may be generated by an alternative device and supplied to thevideo source 1210 for output to display devices. In some embodiments,the video source 1210 may be a Sony® Playstation® 4; this is a devicethat is operable to output video and or game content to two or moredisplay devices. As is shown in FIG. 12, the video source 1210 isoperable to output video to both an HMD 1220 and an additional displaydevice 1230.

The HMD 1220 may be an HMD in accordance with that discussed withreference to FIGS. 1 and 2. Such a device is operable to receive videocontent from the video source 1210, for example via a wired or wirelessdata connection, and display this content to a user.

The additional display device 1230 may be any suitable device that isoperable to display video content to a user. For example, the additionaldisplay device 1230 may be a second HMD (which may or may not be thesame as the HMD 1220), a mobile device, a television (for example, withshutter lenses or polarising glasses for the display of stereoscopiccontent), or a handheld games console. The viewpoint of the user withinthe video content may be dependent upon user inputs using either thedevice 1230 or a controller associated with either the device 1230 orthe video source 1210.

For example, the additional display device 1230 may comprise positionand/or orientation detectors (such as gyroscopes or accelerometers) thatare operable to detect the position and/orientation of the device 1230.This may allow the user to control the viewpoint within the videocontent by moving the device 1230; in embodiments in which the device1230 is an HMD, the control may be performed based on head motion of theuser.

Alternatively, or in addition, if the device 1230 is a handheld consoleor mobile device (for example) then buttons or the like may be providedthat are operable to modify the viewpoint within the content orotherwise control the video content playback (such as pausing thecontent, or taking a screenshot or the like). Similarly, such controlmay be provided in other embodiments. For example, a handheld controllermay be provided in conjunction with and HMD so as to allow user inputsto control the content in conjunction with (or instead of) motion-basedinputs.

In addition to controlling the playback of the video content, suchinputs may also be used to interact with content. For example, if thevideo content relates to a game then the viewer of the additionaldisplay device 1230 may be able to provide inputs to control the game;this may include controlling a character's actions, modifying a virtualenvironment, or otherwise influence the interactive experience of theuser of the HMD 1220.

The additional display device 1230 may be operable to display imagesusing a different aspect ratio to that of images displayed by the HMD1220.

FIG. 13 schematically illustrates a plan view of a virtual environment1300 comprising a user viewpoint 1310 with a corresponding view area1320. The virtual environment 1300 also comprises objects 1330.

The user viewpoint 1310 represents the position in the virtualenvironment 1300 from which a viewpoint is generated for display to thatuser. As discussed above, this may be influenced by user inputs and/or auser's real-world position and/or orientation.

The view area 1320 associated with the user viewpoint 1310 is determinedin a similar manner, and is used to illustrate which features (such asone of the objects 1330) are included within the field of view that ispresented to a viewer.

FIG. 14 schematically illustrates a stereoscopic image pair comprising aleft image 1400 and a right image 1401. The images depict a virtualenvironment, comprising an object 1410 and a character 1420(corresponding to the objects 1330 in FIG. 13, for example), with adisparity that means that when the images are displayed to a viewer inan appropriate manner a three-dimensional effect is provided. Forexample, an HMD may provide an arrangement in which the viewer's lefteye is only able to see the left image and the viewer's right eye isonly able to see the right image.

In general, the images 1400 and 1401 are to be provided to an HMD (suchas the HMD 1220 of FIG. 12) to be displayed in their entirety, with anew image being generated in response to user head motion (or other userinput) so as to ensure that the displayed view matches the user'sexpected or desired view.

Whilst it would be expected that the HMD 1220 would display the images1400 and 1401 in their entirety, the displays associated with theadditional display device 1230 may not be the same size or shape. As aresult of this, a selection of a particular region of each of the images1400 and 1401 may be performed so as to generate appropriate images fordisplay at the additional display device 1230.

FIG. 15 schematically illustrates an embodiment in which respectiveportions 1500 and 1501 of the generated images 1400 and 1401 areselected for display. In some embodiments, this is performed so as toindicate which portions of the generated images 1400 and 1401 are to bedisplayed by an additional display device 1230. The additional displaydevice 1230 may be operable to select image portions 1500 and 1501 fordisplay in dependence upon a viewer input and/or motion tracking of theviewer, for example.

The display of only a portion of generated images 1400/1401 may beadvantageous in that a user is able to pan about an image freely withoutrequiring additional images to be generated to represent the imagecontent that was outside of the initially-displayed image. For example,a user may wish to see the whole of the tree 1410 in the image 1401, andas a result move the selected portion 1501 to the left. This imagecontent already exists, and so a portion of the image 1401 to the leftof the area 1501 may be displayed without generating new image content.A delay in providing the correct image content for displaying aparticular view may therefore be reduced, as it may not be necessary togenerate new image content in response to a request for an updatedviewpoint.

In some embodiments, the selected areas 1500 and 1501 have the sameaspect ratio as the initially generated images 1400 and 1401 (commonly16:9, but not limited to this). This may be the case if the display ordisplays associated with the additional display device 1230 are of thesame aspect ratio as those of the HMD 1220, for example. The selectedareas 1500 and 1501 may be enlarged for display by the additionaldisplay device 1230 if the displays are of a greater resolution than theselected areas 1500 and 1501, or may be displayed without adjustment ifthe displays are of the same or similar resolution to that of theselected areas 1500 and 1501.

In other embodiments, the selected areas 1500 and 1501 are of adifferent aspect ratio to that of the initially generated images 1400and 1401. This may be advantageous in the case that the aspect ratio ofthe additional display device 1230 differs from that of the HMD, forexample. In an embodiment in which the additional display device 1230 isa mobile phone, a single high definition display (such as a 1920×1080pixel resolution) may be provided that is required to show both images;this results in two separate areas each with 960×1080 pixels. This iseffectively an aspect ratio of 8:9, rather than the 16:9 of the fullscreen and the generated image (although the generated image may be anyother suitable aspect ratio, of course).

In such a case, it may be advantageous to select a different aspectratio for the selected images 1500 and 1501 to those of the initialimages 1400 and 1401, and then perform any scaling or the like asrequired.

The selected areas 1500 and 1501 may be additionally or alternativelyselected to reduce apparent motion in the displayed scene; if the imagescorrespond to those presented to the user of the primary HMD, then theseimages are likely to track micro-motions of that user's head, and othersmall motions associated with their breathing, arm movements etc.

These could be disorientating or cause discomfort for a secondary user,as their display appears to react to background minor head movementsthat are not their own.

Consequently, an image stabilisation technique may be used to removesmall movements between frames by adjusting the position of the selectedareas in a direction opposite to that of the detected movement in thesupplied images. Because the selected areas 1500 and 1501 are smallerthan the source images, this movement is possible within a limitedrange, but that range is likely to accommodate most small andinvoluntary movements made by the user of the primary HMD.

It will be appreciated that head motions of the user of the secondarydisplay may then also be factored in to the selection of the areas, sothat they still perceive changes in view in response to their ownvoluntary (or possibly involuntary) head motion, but with respect to asubstantially image-stabilised view of the source images.

As an alternative to selecting the areas 1500 and 1501 at the additionaldisplay device, the selection of the areas 1500 and 1501 from the images1400 and 1401 may be performed by the video source 1210 such that onlythe areas 1500 and 1501 are transmitted to the additional display device1230. Alternatively, the areas 1500 and 1501 may be selected by thevideo source 1210 and identified when transmitting the images 1400 and1401 to the additional display device 1230. As a further alternative,the images 1400 and 1401 may be transmitted to the additional displaydevice 1230, and the additional display device 1230 performs processingto identify and then display areas 1500 and 1501.

Such an arrangement is therefore operable to provide content to aplurality of devices simultaneously, such that additional viewers mayhave an immersive experience corresponding to that of the user of theHMD 1220. Further modifications may further improve the spectatorexperience, as is described below, by providing further reductions tothe processing requirements (relative to generating separatestereoscopic streams) and/or reducing the delay in providing appropriateimages to a viewer.

In some embodiments, a single image of the stereoscopic image pair maybe output to an additional display device 1230 if that device is notoperable to display a pair of images to provide stereoscopic imagecontent. Alternatively, the additional display device 1230 may be ableto perform processing to extract and display only a single image of asupplied stereoscopic image pair.

FIG. 16 schematically illustrates the virtual environment 1300 of FIG.13 with an additional viewpoint 1600 having a corresponding view area1610. The viewpoint 1310 has associated view areas 1320 and 1321.

The additional viewpoint 1600 corresponds to that of a user of theadditional display device 1230. The location of this viewpoint may bedetermined in accordance with any of the above methods, or may bedetermined by the video source in dependence upon the content (forexample). In some embodiments a view may represent a spectator of agame, a further participant in a game, or a further viewer of videocontent. Of course, any number of additional viewpoints 1600 may beprovided in accordance with the requests of one or more spectators.

The view area 1610 corresponding to the viewpoint 1600 encompasses amuch wider field of view than those view areas 1320 and 1321 associatedwith the viewpoint 1310. While the differences between the respectiveview areas are shown to be extremely large, this may be an exaggerationin some embodiments; for example, a second display device may not beable to display such a large field of view due to screen size or thelike. The relative size of the view areas here are selected so as toillustrate that the view area 1610 associated with the additionalviewpoint 1600 is wider than that of a view area associated with theviewpoint 1310.

The use of a wider field of view for an additional viewpoint isadvantageous in that it reduces the likelihood of a user changing therequested viewpoint, as they are already able to see a large number ofobjects. FIG. 16 illustrates this in that only one of the objects 1330may be viewed at a time from the position of the viewpoint 1310, whilstthe view area 1610 encompasses both of the objects 1330 without the useradjusting the viewpoint further. In some embodiments the size, shape, ororientation of the view area 1610 may be determined so as to include all(or a significant proportion) of the objects of interest within a scene.For example, the view area 1610 may be defined so as to encompass all‘quest objects’ in a game scene in addition to non-playable characterswith which a player is likely (or at least able) to interact.

By reducing the frequency with which the user is likely to change theviewpoint, processing required to generate new viewpoints may bereduced. In addition to this, the importance of providing a smoothtransition may be reduced; an equivalent change in view direction mayrepresent a much greater change in content when the field of view isnarrow than when it is wide. This is because the change in direction ismuch smaller relative to the size of the field of view for a wide viewarea; for example, a head rotation of 10 degrees represents a third of aview area of 30 degrees, but only a tenth of a view area of 100 degrees.

In view of this consideration, fewer new images may be generated by thevideo source 1210 for display by the additional display device 1230, forexample, which may reduce the processing burden on the video source 1210and/or additional display device 1230.

As a summary of the above examples (the embodiments described withreference to FIG. 15 or 16), it is apparent that the or each imagedisplayed at one of the first display device and the second displaydevice is a subset of the or each image displayed at the other of thefirst display device and the second display device, and that theviewpoint of the display device displaying a subset of the or each imageis operable to modify the displayed field of view independently of thefield of view displayed at the other display device.

FIG. 17 shows an embodiment in which wide field of view content isgenerated for supply to an additional display device 1230, but only aportion of the wide field of view content is actually displayed by theadditional display device 1230 at any given time. Alternatively, anarrangement may be provided in which the wide field of view content isgenerated, but only a portion of the content is transmitted to theadditional device in dependence upon the indicated viewpoint associatedwith the device.

FIG. 17 schematically illustrates a view area 1620 representing theportion of the virtual environment 1300 that is displayed to the user.The view area 1620 is present in the view area 1610 corresponding to theadditional viewpoint 1600.

A reduced frame rate of supplied content may be acceptable in such anarrangement as if a user pans within the content image content alreadyexists that may be used; therefore the likelihood of a user panning andthere not being sufficient image content to display a full image isreduced. The refresh rate of the display may be selected to besufficiently high such that the image portions are updated with a highenough frequency so as to provide responsive viewpoint changes.

Of course, the view area 1620 may still comprise a wider field of viewthan a corresponding view area associated with a viewpoint of an HMDuser, in line with the discussion above with reference to FIG. 16. Thebenefits of a displayed wide field of view in reducing the likelihood ofa change of viewpoint may therefore be obtained even when onlydisplaying a portion of the generated content.

In some embodiments, the size of the view area may be defined independence upon the content being provided by the video source. Forexample, the size of the view area may be defined so as to encompass allobjects of interest (such as non-playable characters and interactiveobjects) as these are the objects that are most likely to attract aplayer's attention. By providing a view that includes all of (or asubstantial portion of) these objects the desire of the viewer to changeview position may be reduced significantly.

Alternatively, or in addition, the viewpoint may be defined so as tocomprise a main player (such as that controlled by the user of the HMD1220 in FIG. 12). This may be particularly appropriate in arrangementsin which the additional display device 1230 is used to provide aspectator view of the HMD 1220 user's gameplay.

Video content for the additional display device 1230 may be generated inany suitable manner. In some examples, game data or other contentgeneration data may be used to render content for the additional displaydevice 1230. Alternatively, or in addition, content generated fordisplay by the HMD 1220 may be provided to the additional displaydevice.

In some embodiments, an application is provided either at the videosource 1210 or the additional display device 1230 that is operable toaccess the rendering resources used by the video source 1210 to generatecontent for display at the HMD 1220. This may comprise accessing thez-buffers, render lists and texture information stored by the videosource 1220, for example.

Using these resources, the application is operable to generate a viewfor display by the additional display device 1230 by performing its ownrendering process. In some embodiments, the video source 1210 isoperable to maintain z-buffers, render lists and texture information (orany other type of information that may be used for rendering a virtualscene) for a greater portion of a virtual scene than is to be displayedat the HMD 1220; this is an example of a modification to the processthat may ensure that a wider field of view may be generated for thespectator.

Arrangements such as those described above should not be limited toembodiments in which VR content is provided to a HMD user; the teachingshere may be equally applicable to arrangements in which non-immersivetwo-dimensional content is provided. This may provide a more interactiveand intuitive spectator experience, for example, for any number of gamesor other types of content.

FIG. 18 schematically illustrates a content generation and displaysystem. This system comprises an image generation unit 1800, an imagetransmission unit 1810, a first image display unit 1820 and one or moreadditional display units 1830.

The image generation unit 1800 is operable to generate one or moreimages of a virtual environment for display at a first display deviceand a second display device. The image generation unit 1800 is operableto generate one or more images in dependence upon a viewpoint in thevirtual environment associated with a user of the first display device(such as the HMD 1220). In some embodiments, the image generation unit1800 is operable to generate one or more stereoscopic image pairs.

The features of the image generation unit 1800 are described in moredetail below with reference to FIGS. 19 and 20.

The image transmission unit 1810 is operable to transmit generatedimages to each of the first display device 1820 and an additionaldisplay device 1830. In some embodiments, the image transmission unit1810 is operable to transmit generated images to two or more additionaldisplay devices 1830. The image transmission unit 1810 may utilise anysuitable wireless or wired communication method in order to transmitimages, for example an HDMI cable, USB cable, WLAN connection or aBluetooth® connection.

In embodiments in which the image generation unit 1800 is operable togenerate one or more stereoscopic image pairs, the image transmissionunit 1810 may be operable to transmit a single image of a generatedstereoscopic image pair to the additional display device 1230.

As noted above, in some embodiments of the presently disclosedarrangement it may be possible to provide content to an additionaldisplay device 1230 with a reduced frame rate without a significant lossof immersion for a user. In such embodiments, the image transmissionunit is operable to transmit fewer images to the additional displaydevice 1230 than to the first display device (such as the HMD 1220).

The first image display unit 1820 is the primary display unit fordisplaying content generated by the image generation unit 1800. This maycorrespond to a display associated with the HMD 1220 of FIG. 12, forexample, and the first image display unit 1820 may generally beassociated with a user regarded as the main player of a game or thelike. In some embodiments, the first display device is a head-mountabledisplay device.

The additional image display unit(s) 1830 correspond to the additionaldisplay device 1230 of FIG. 12. The additional image display unit(s) maybe display units associated with any one of a head-mountable displaydevice, a mobile device (such as a mobile phone or portable gamesconsole), a monitor, and a television. Each additional image displayunit 1830 is operable to display a portion of each received image; inthe examples described above, this may correspond to displaying an imagearea 1500 of a received image 1400 (see FIG. 15). In some embodiments,the additional image display unit 1830 is operable to select an imageportion for display in dependence upon a viewer input and/or motiontracking of the viewer.

As would be apparent from consideration of the above description, imagesmay be generated and transmitted in the form of stereoscopic pairs(although this is not essential) such that three-dimensional images maybe viewed by a viewer.

FIG. 19 schematically illustrates an image generation unit 1800 thatcomprises a viewpoint identification unit 1900, an image dataidentification unit 1910 and a first image generation unit 1920.

The viewpoint identification unit 1900 is operable to identify aviewpoint within the virtual environment that comprises a location,orientation and field of view (or any other suitable method for defininga viewpoint). These elements may be used to determine which portions ofthe virtual environment should be displayed to a viewer.

The image data identification unit 1910 is operable to identify the datarequired to generate an image for display according to that determinedby the viewpoint identification unit 1900. For example, this maycomprise locating meshes and/or textures corresponding to objects thatwill be visible within the generated image. This identification may alsocomprise locating where the image data is stored, for example in a localstorage device or on a server that stores the image data.

The first image generation unit 1920 is operable to generate one or moreimages using the information from the viewpoint identification unit 1900and the data identified by the image data identification unit 1910. Thismay comprise any suitable rendering process or processes, such asperforming z-culling, light modelling (such as ray tracing), and texturemapping. The first image generation unit 1920 may be further operable togenerate stereoscopic image pairs for transmission, for example byconsidering depth information associated with the objects in the virtualscene to generate a pair of images with the correct disparity offsetfrom one another.

FIG. 20 schematically illustrates an alternative image generation unit1800, for use when separate images are generated for the first displaydevice and the additional display devices. This image generation unit1800 comprises a viewpoint identification unit 2000, image dataidentification unit 2010, first image generation unit 2020 and secondimage generation unit 2030.

In some embodiments, the image generation unit shown in FIG. 20 operatesin a manner according to that of FIG. 19, in which each of the unitsperforms the function of the corresponding unit. The second imagegeneration unit 2030 performs the additional function of generatinglower-resolution versions of the images generated by the first imagegeneration unit 2020, for example, or identifies which images may beomitted from the transmission to an additional display device. Thesecond image generation unit 2030 may also perform image manipulationsin images generated by the first image generation unit 2020, such ascropping or otherwise resizing the images. Of course, the first andsecond image generation units 2020 and 2030 may be formed as a singleimage generation unit.

In some embodiments, it is therefore apparent that the image generationunits 2020 and 2030 (or the combined single image generation unit) areoperable to generate low-resolution and high-resolution versions of thesame images, and the image transmission unit 1810 is operable totransmit the high-resolution images to the first display device 1220 andthe low-resolution images to the additional display device 1230. Theresolution of the low-resolution images may be selected in dependenceupon the display properties of the additional display device 1230, suchthat an appropriate-resolution image is provided to each additionaldisplay device 1230.

In some embodiments, it is therefore apparent that the image generationunits 2020 and 2030 (or the combined single image generation unit) areoperable to generate reduced-size versions of the one or more firstimages, and the image transmission unit 1810 is operable to transmit thereduced-size images to the additional display device 1230.

In some embodiments in which the image generation unit of FIG. 20 isused, the first image generation unit is operable to generate one ormore first images of a virtual environment for display at a firstdisplay device, and the second image generation unit is operable togenerate one or more second images of the virtual environment fordisplay at a second display device, the one or more second images havinga larger field of view than the one or more first images. This may allowimages to be generated in accordance with the discussion relating toFIGS. 16 and 17, for example.

In such embodiments, the viewpoint identification unit 2000 is operableto identify two or more viewpoints (one for a first user, and one ormore corresponding to one or more additional users). It should be notedthat there need not be a one-to-one mapping between viewpoints andusers, as multiple users may share the same viewpoint, for example, oradditional viewpoints may be identified at which to generate imagecontent for future playback or the like.

In some embodiments, the viewpoints associated with the one or moreadditional users are identified with a wider field of view by theviewpoint identification unit 2000. The breadth of the field of view maybe determined either using fixed values (for example, set by a user or acontent provider), or by using knowledge of the virtual scene so as toensure a sufficient number of significant objects are to be includedwithin a generated image, for example. Alternatively, or in addition,the breadth of the field of view may be determined in dependence upon acorresponding user's view history; for example, a user with a history ofnot changing view direction often may be provided with an image of anarrower field of view than that provided to a user that is prone tochange view direction often.

The image data identification unit 2010 is operable to identifyresources in the same manner as the image data identification unit 1910of FIG. 19, for each of the identified viewpoints.

The first image generation unit 2020 is operable to generate images inthe same manner as the first image generation unit 1920 of FIG. 19. Thisimage generation is performed for a first viewpoint, generally thatcorresponding to the first image display device.

The second image generation unit 2030 is operable to generate images fora second display device. This is performed in a similar manner as therendering performed by the first image generation unit 2020, althoughfor a different viewpoint and generally so as to generate an image witha wider field of view (as described above).

The first and second image generation units 2020 and 2030 may instead beformed as a single unit in some embodiments. This unit may be located atthe image source 1210 of FIG. 12, for example. Alternatively, the firstand second image generation units 2020 and 2030 may be located at therespective display devices (for example, the HMD 1220 and additionaldisplay device 1230 of FIG. 12) in embodiments in which the dataidentified by the image data identification unit 2010 is transmitted tothe display devices instead of generated images.

Of course, in embodiments in which more than one additional displaydevice is used there may be a third or even greater number of imagegeneration units. In some embodiments, an image generation unit isprovided for each display device. In some embodiments, an imagegeneration unit is provided for each desired viewpoint.

FIG. 21 schematically illustrates a content generation and transmissionmethod.

A step 2100 comprises generating one or more images of a virtualenvironment for display at a first display device. Examples of imagegeneration methods are described with reference to FIGS. 22 and 23below.

A step 2110 comprises transmitting generated images to each of the firstdisplay device and an additional display device. This may be performedusing any suitable wired or wireless communication method.

FIG. 22 schematically illustrates an image generation method.

A step 2200 comprises identifying a viewpoint in the virtualenvironment. As noted above, identifying a viewpoint within the virtualenvironment comprises determining a location, orientation and field ofview (or any other suitable method for defining a viewpoint). Theseelements may be used to determine which portions of the virtualenvironment should be displayed to a viewer.

A step 2210 comprises identifying image data corresponding to theviewpoint identified in step 2200. For example, this may compriselocating meshes and/or textures corresponding to objects that will bevisible within the generated image. This identification may alsocomprise locating where the image data is stored, for example in a localstorage device or on a server that stores the image data.

A step 2220 comprises generating one or more first images using theidentified image data; as described above, this may include any suitablerendering process or processes, such as performing z-culling, lightmodelling (such as ray tracing), and texture mapping. In addition tothis, the step 2220 may comprise generating stereoscopic image pairsrather than single images, for example by considering depth informationassociated with the objects in the virtual scene to generate a pair ofimages with the correct disparity offset from one another.

A step 2230 is an optional step that may be performed, and it comprisesthe generation of one or more second images using either the identifiedimage data from step 2210 or the images generated in step 2220. In theformer case, this comprises rendering an image separately from the oneor more first images while in the latter case a manipulation of the oneor more first images is performed as appropriate. In some embodiments,one or more second images may be generated either aslow-quality/low-resolution versions of the first images, orcropped/resized versions of the same image.

FIG. 23 schematically illustrates an alternative image generation methodto that of FIG. 22.

A step 2300 comprises identifying a plurality of viewpoints in thevirtual environment; one corresponding to a first user, and one or morecorresponding to one or more additional users. In some embodiments, theviewpoints associated with the one or more additional users areidentified with a wider field of view.

A step 2310 comprises identifying image data corresponding to the one ormore viewpoints, in the same manner as the step 2210.

A step 2320 comprises generating one or more first images for display bya first display device, in the same manner as the step 2220.

A step 2330 comprises generating one or more second images correspondingto an additional display device. This is performed in a similar manneras the rendering performed by the first image generation unit 2020,although for a different viewpoint and generally so as to generate animage with a wider field of view.

It should be understood that the step 2330 may be repeated any number oftimes to generate further images, such as to generate one or more thirdimages corresponding to a third display device or viewpoint.

FIG. 24 schematically illustrates a content display method.

A step 2400 comprises receiving image content from an image source; forexample, the one or more second images generated by the methodillustrated by FIG. 22.

A step 2410 comprises selecting an image portion for display. Thisportion is a part of a received image, for example the image area 1500of the image 1400 in the example of FIG. 15.

An appropriate portion for display may be determined in dependence upona viewer input and/or motion tracking of the viewer, for example. Thismay be determined prior to transmission of the image content (such thatonly the selected image portion may be transmitted to the displaydevice), or the display device may perform the selection after receivingthe image content.

A step 2420 comprises displaying the selected image portion, as selectedin the step 2410.

It will be appreciated that example embodiments can be implemented bycomputer software operating on a general purpose computing system suchas a games machine. In these examples, computer software, which whenexecuted by a computer, causes the computer to carry out any of themethods discussed above is considered as an embodiment of the presentdisclosure. Similarly, embodiments of the disclosure are provided by anon-transitory, machine-readable storage medium which stores suchcomputer software.

It will also be apparent that numerous modifications and variations ofthe present disclosure are possible in light of the above teachings. Itis therefore to be understood that within the scope of the appendedclaims, the disclosure may be practised otherwise than as specificallydescribed herein.

The foregoing discussion discloses and describes merely exemplaryembodiments of the present invention. As will be understood by thoseskilled in the art, the present invention may be embodied in otherspecific forms without departing from the spirit or essentialcharacteristics thereof. Accordingly, the disclosure of the presentinvention is intended to be illustrative, but not limiting of the scopeof the invention, as well as other claims. The disclosure, including anyreadily discernible variants of the teachings herein, defines, in part,the scope of the foregoing claim terminology such that no inventivesubject matter is dedicated to the public

1. A content generation system comprising: an image generation unitoperable to generate one or more images of a virtual environment fordisplay at a first display device and a second display device; and animage transmission unit operable to transmit generated images of thevirtual environment to each of the first display device and the seconddisplay device, wherein the or each image displayed at one of the firstdisplay device and the second display device is a subset of the or eachimage displayed at the other of the first display device and the seconddisplay device, and wherein the viewpoint of the display devicedisplaying a subset of the or each image is operable to modify thedisplayed field of view independently of the field of view displayed atthe other display device.
 2. A content generation system according toclaim 1, wherein the first display device is a head-mountable displaydevice.
 3. A content generation system according to claim 1, wherein thesecond display device is one of a head-mountable display device, amobile device, a handheld games console, a monitor, and a television. 4.A content generation system according to claim 1, wherein the seconddisplay device is operable to select a viewpoint for display independence upon a viewer input and/or motion tracking of the viewer. 5.A content generation system according to claim 1, wherein the seconddisplay device is operable to display images using a different aspectratio to that of images displayed by the first display device.
 6. Acontent generation system according to claim 1, wherein the imagegeneration unit is operable to generate the one or more images independence upon a viewpoint in the virtual environment associated with auser of the first display device.
 7. A content generation systemaccording to claim 1, wherein the image transmission unit is operable totransmit fewer images to the second display device than to the firstdisplay device.
 8. A content generation system according to claim 1,wherein the image transmission unit is operable to transmit generatedimages to two or more additional display devices.
 9. A contentgeneration system according to claim 1, wherein: the image generationunit is operable to generate low-resolution and high-resolution versionsof the same images; and the image transmission unit is operable totransmit the high-resolution images to the first display device and thelow-resolution images to the second display device.
 10. A contentgeneration system according to claim 9, wherein the resolution of thelow-resolution images is selected in dependence upon the displayproperties of the second display device.
 11. A content generation systemaccording to claim 1, wherein: the image generation unit is operable togenerate reduced-size versions of the one or more first images; and theimage transmission unit is operable to transmit the reduced-size imagesto the second display device.
 12. A content generation system accordingto claim 1, wherein: the image generation unit is operable to generateone or more stereoscopic image pairs; and the image transmission unit isoperable to transmit a single image of a generated stereoscopic imagepair to the second display device.
 13. A content generation methodcomprising: generating one or more images of a virtual environment fordisplay at a first display device and a second display device; andtransmitting generated images of the virtual environment to each of thefirst display device and the second display device, wherein the or eachimage displayed at one of the first display device and the seconddisplay device is a subset of the or each image displayed at the otherof the first display device and the second display device, and whereinthe viewpoint of the display device displaying a subset of the or eachimage is operable to modify the displayed field of view independently ofthe field of view displayed at the other display device.
 14. (canceled)15. A non-transitory machine-readable storage medium which storescomputer software, which when executed by a computer, causes thecomputer to carry out actions, comprising: generating one or more imagesof a virtual environment for display at a first display device and asecond display device; and transmitting generated images of the virtualenvironment to each of the first display device and the second displaydevice, wherein the or each image displayed at one of the first displaydevice and the second display device is a subset of the or each imagedisplayed at the other of the first display device and the seconddisplay device, and wherein the viewpoint of the display devicedisplaying a subset of the or each image is operable to modify thedisplayed field of view independently of the field of view displayed atthe other display device.